Skip to content

stats: where scraper, bot, and scanner traffic comes from - #1476

Merged
aaylward merged 2 commits into
mainfrom
stats-geo
Sep 2, 2026
Merged

stats: where scraper, bot, and scanner traffic comes from#1476
aaylward merged 2 commits into
mainfrom
stats-geo

Conversation

@aaylward

@aaylward aaylward commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Closes #1467, part of #1365. Each request's client address is placed in a country and geo_stats keys day, host, agent class, and country with request, 403, and probe counts, served at /stats/v1/countries. The database is DB-IP's free country CSV (CC BY 4.0; the stats page on muchq.com carries the attribution), pinned as an http_file by URL and sha256 and bundled into the stats image at /geo, where the service loads it at boot into a sorted range table and binary-searches it rather than pulling in a GeoIP library. A new month is a pin bump; a file that will not load means every row reads -- with an error in the log, never a boot failure. Hosting a fallback copy of the pinned file is #1478.

Panel ran against the first push and its findings are the second commit; the third moves the file from a hand-uploaded S3 object to the image. Tested with bazel test on stats (store tests against a local postgres:18 on the second commit; the third touches no SQL), the deploy config, and the image build.

Each request's client address is placed in a country from DB-IP's free
country CSV, loaded at boot from the stats bucket into a sorted range
table — no library, a binary search is the whole lookup — and geo_stats
keys day, host, agent class, and country with request, 403, and probe
counts. Served at /stats/v1/countries. GEO_DB_KEY names the object;
unset, every row reads "--", and a key that fails to load fails boot.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 2, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
1d4-web 8faee61 Commit Preview URL

Branch Preview URL
Sep 02 2026, 01:51 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 2, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
ast-lol 8faee61 Commit Preview URL

Branch Preview URL
Sep 02 2026, 01:53 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 2, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
iili 8faee61 Commit Preview URL

Branch Preview URL
Sep 02 2026, 01:52 PM

A geo key that will not load is retried while the bucket wakes up, then
logged and run without — an all-"--" table, never a boot failure that
takes the other endpoints with it. Ranges are validated after unmapping,
overlaps lose to the range they sit in, and country codes are interned so
the table does not pin its CSV lines. Tests cover the refused probe, the
unplaced row, and the boot path; the deploy test pins GEO_DB_KEY; the
READMEs say which prefixes the IAM user needs and where the file goes.
@aaylward
aaylward merged commit acd4431 into main Sep 2, 2026
22 checks passed
@aaylward
aaylward deleted the stats-geo branch September 2, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stats pipeline: geo breakdown of scraper, bot, and probe traffic

1 participant